An automatic method for assessing the teaching impact of books from online academic syllabi

Scholars writing books that are widely used to support teaching in higher education may be undervalued due to a lack of evidence of teaching value. Whilst sales data may give credible evidence for textbooks, it may poorly reflect educational uses of other types of books. As an alternative, this article proposes a method to automatically search for mentions of books in online academic course syllabi based on Bing searches for syllabi mentioning a given book, filtering out false matches through an extensive set of rules. The method had an accuracy of over 90% based on manual checks of a sample of 2,600 results from the initial Bing searches. Over a third of about 14,000 monographs checked had one or more academic syllabus mention, with more in the arts and humanities (56%) and social sciences (52%). Low but significant correlations between syllabus mentions and citations across most fields, except the social sciences, suggest that books tend to have different levels of impact for teaching and research. In conclusion, the automatic syllabus search method gives a new way to estimate the educational utility of books in a way that sales data and citation counts cannot.